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Introduction 

This year the Cellular Systems Division of Motor- 
ola submitted an application to the IEEE Computer 
Society for a Software Process Achievement 
Award. We placed second overall, with the Award 
going to our hosts, the Software Engineering Labo- 
ratory. In our application for the award we made 
public results of more than five years of effort we 
have been undertaking to improve our software pro- 
cesses. 

Like many large software development 
organizations, we have experienced our share of 
customers who complain about product defects, 
failure to meet schedule commitments, and our 
inability to provide the software functionality they 
are demanding. By early 1990, the staff had come to 
recognize that the software processes in place were 
inadequate to meet our customer needs. Thus, in 
1990 we began using the SEI Process Maturity 
Model (PMM) and Humphrey's Managing the 
Software Process 1 to help us define the 
requirements for a more mature software process. 
Our ultimate goals were (and still are) to improve: 
our customer's satisfaction, 

- our product quality, 

our on-time delivery record, and 
our productivity. 

In April of 1991, a team of SEI-trained assessors, 
both from SEI and from across Motorola, assessed 
our organization at Level 1. Then in late 1991 we 
were presented with a classic example of “require- 
ments creep” when the SEI announced their first 
draft version Capability Maturity Model (CMM) 2 
which was intended to replace the PMM. Careful 
review lead us to conclude that we had no choice 
but to adapt this more rigorous and detailed set of 
process requirements. We found to our delight that 
the software process architecture we developed, 
which was implicit in IEEE Std 1074-1991 “Stan- 
dard for Developing Software Life Cycle Pro- 
cesses,” 3 was robust enough to meet the new CMM 


requirements. What needed attention were the “pro- 
cess specifications.” These would have to be far 
more detailed to assure conformance to the CMM 
requirements. We had previously formed working 
groups to write process specifications for all pro- 
cesses, and now we began to identify the changes 
needed to meet the new CMM requirements. Next, 
we prioritized our efforts based on the CMM five- 
level model. 

In June of 1993, after months of implementing this 
Software Process Improvement (SPO Plan, we were 
re-assessed formally (using the PMM) at Level 2. 
More importantly, as more of our processes have 
begun to conform to the CMM requirements, we 
have begun to demonstrate significant measurable 
improvement in delivered product quality and on- 
time delivery, delivering more functionality to a 
more-satisfied customer, as accompanying data will 
support. Our data gathering activities have lagged 
behind other process changes, and key process mea- 
sures were not routinely made before 1992, but we 
think that it is important to keep in mind that the 
data presented covering the last six quarters effec- 
tively represent results of process improvements 
underway since early 1991. 

To support the Nomination of the SPI team at CSD 
a set of representative data was prepared. We pre- 
sented data from a single product software develop- 
ment group representing about three hundred 
developers in our division. Since the submission of 
this application we have continued our efforts, and 
new data continues to demonstrate the benefits. We 
will review all of the data we have available to us at 
this time, which represents the time frame from the 
first quarter of 1992 to the end of the second quarter 
of 1994. Data from all projects completed by this 
product group and released to customers in that 
time frame are included. Six charts will be pre- 
sented. 

Figure 1 

This figure shows our progress made in achieving 
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compliance with the requirements of the six Level 
2 Key Process Areas (KPAs) named in the SEI 
CMM, Requirements Management (RM), Project 
Hanning (PP), Project Tracking (PT), 
Subcontractor Management (SM), Quality 
Assurance (QA), and Configuration Management 
(CM). 


An internally-developed procedure is used to assess 
compliance, and each development group conducts 
quarterly internal self-assessments. 4 The 
assessment procedure focuses on key practices 
described in the CMM, and compliance is 


co ntin gent upon evidence of the presence of each 
key practice. The “percent compliance” described 
in this Figure is therefore the mean percent 
compliance of all of the key practices in each KPA 
which are evident to the assessment team. Outside 
team members from other development 


organizations and from the software quality 
assurance organization participate in these 
assessments to assure more-uniform and rigorous 


scoring. 


The first round of these assessments was held in the 
third quarter of 1992, and the results of that 
assessment are compared to the current scores. The 
entire development organization was assessed at 
Level 2 using the PMM in June of 1993, but this 
development group had not yet achieved complete 
compliance with all of the requirements of the 
CMM at that time. However, since then significant 
progress has been made, and full implementation of 
all the key practices described in each KPA is now 
evident. 


Figure 2 

With the completion of our formal Self-assessment 
in June 1993, when we were rated at Level 2, the 
entire organization has moved forward with an 
initial assessment of our status with respect to the 
key practices found in Level 3 KPAs using our self- 
assessment procedure. The initial scores of this 
development group are presented in this chart. The 
initial conclusion one might draw from this chart is 
that die group is far from compliant with the 
requirements for Level 3. In view of our initial 
scores on the Level 2 Key Process Areas, however, 
we are confident that the group can be expected to 
make rapid progress toward compliance. Combined 


with the information presented in Figure 1, we can 
see that the group is in full compliance with Level 
2 KPAs, and working on improvements on the 
Level 3 KPAs. 


Figure 3 

A customer survey is conducted regularly by an 
independent market research firm using a 
“Motorola Confidential Proprietary” survey 
questionnaire. In constructing this survey 
questionnaire “Key Drivers” have been identified 
which represent our effort to measure what our 
customers think is important Each satisfaction 
survey measures our performance on these Key 
Drivers. Figure 3 compares our percent 
improvement in this product group for the Key 
Drivers which are concerned with software, in 
comparison to our performance in 1991. Since the 
survey contents and results are confidential, we 
have represented our progress by means of an 
index, with year-end 1991 results being “1,” and 
year-end 1992 and 1993, and year-to-date 1994 
being shown relative to that index quantity. 

Figure 4 

To explain Figure 4, some specific definitions are 
required. 

Customer Found Defects are those post-release 
defects which are found by the customer. This does 
not include post-release defects found by Motorola 
internally, or defects of which customers have been 
notified before these customers find them. 

Each customer found defect is recorded based upon 
the release in which it is found. A “window of 
opportunity” to find defects exists for each 
successive release. For a particular release, the first 
opportunity to find and report defects occurs at the 
time the first customer installs it. Defects in that 
release can be found by customers up to the time the 
last customer using that release retires it. Most 
releases are in use about 12 to 18 months. When a 
release is made in a particular quarter, and defects 
are reported against that release, the number of 
Customer Found Defects for all releases in that 
quarter is incremented. Over time, if additional 
defects against that release are reported, the 
quarterly total of defects for releases in that quarter 
is incremented. As releases are retired, since defects 
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can no longer be repotted further against them, the 
total defect count becomes fixed. Our experience, 
like most software developers, is that most 
Customer Found Defects ever found are reported in 
the first quarter of use. 

Delta KAELOC is the size of the added, deleted, 
and modified source code expressed in thousands of 
Assembly-Equivalent Lines of Code.This number 
is calculated based on a factor specific to each 
programming language used using the table 
provided by Capers Jones of SPR, Inc. 

Total KAELOC is the total size of the released 
software expressed in thousands of Assembly- 
Equivalent Lines of Code. This number is 
calculated based on a factor specific to each 
programming language used. 

Figure 4 demonstrates that in this time period the 
number of customer found defects has continued to 
decline, and that our most-recent releases are 
approaching 6 sigma quality. 

Figure 5 

Delay in delivery of promised software releases is a 
key contributor to customer dissatisfaction. In all of 
our product groups, release dates are forecast at the 
time of “project initiation” when the release project 
plan is approved and development begins. Figure 5 
records for each release in a quarter how long after 
the forecasted release date the actual release 
occurred. Coincidentally, there has been one release 
per quarter for this product for the last two years. 

Figure 5 shows a step-function improvement 
occurred in on-time deliveries between the releases 
in the second and third quarters of 1992. This came 
about primarily through better management 
controls in project planning and project tracking. 
Demonstrating that we are still a Level 2 
organization, one release was delayed significantly 
in the second quarter of 1993 because of a delay in 
delivery of a vendor’s code, and because some key 
staff members were temporarily reassigned to 
another project. In the fourth quarter of 1993 
another release was delayed because of extended 
negotiations with a key customer on feature content 
for the release. This experience clearly highlights 
why both subcontractor management, project 
tracking, and requirements are key contributors to 


customer satisfaction. A note of explanation about 
the seeming lack of data for the first quarter of 
1993. In fact, this release was exactly on time, thus 
the delay was zero months. 

Figure 6 

More and more functionality is being demanded by 
our customers, and with each new release we place 
more functionality into the customer’s hands. 
Figure 6 demonstrates the extent to which the 
amount of new code (Delta KAELOC, as defined in 
the note to Figure 4) is growing at each release. In 
data not presented here we have measured that our 
productivity in terms of the number of lines of code 
produced by each software engineer has more than 
doubled in this time. Thus, while we have added 
staff, the staff has continued to increase the amount 
of code being delivered. The decline in the total 
number of new lines of code evident in 1994 results 
from the fact that this product development group is 
in the midst of a major product upgrade this year 
and only small, point releases have been made this 
year while most work continues to focus on the 
planned major upgrade to occur in the first quarter 
of 1995. 

Returning to a topic mentioned in the note to Figure 
4 we want to reiterate that even though we have 
increased the number of lines of code delivered 
with each new release by seven-fold, we are still 
seeing a significant decline in the number of 
customer-found defects in these releases. Stated 
simply, we are releasing more functionality to our 
customers, with higher productivity, and with fewer 
defects. 

Summary 

We believe that the key success elements are related 
to our recognition that Software Process Improve- 
ment (SPI) can and should be organized, planned, 
managed, and measured as if it were a project to 
develop a new process, analogous to a software 
product. In summary, we believe that our process 
improvements have come as the result of these key 
elements: 

use of a rigorous, detailed requirements set 

(CMM), 

use of a robust, yet flexible architecture (IEEE 

1074), 
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use of a SPI project, resourced and managed 
liiff: other work, to produce the specifications 
and implement them, and 
development of both internal and external 
goals, with metrics to support them. 

We have achieved significant, measurable results as 
a result of these efforts, and we want to share these 
findings with a broad industry audience. Our efforts 
may be viewed as unique in the sense that our 
business is entirely commercial and we have no 
customer pressure to adopt any particular paradigm 
for improvement, yet we selected the SEI Process 
Maturity Model and have successfully used the 
requirements of this Model to drive software 
process improvements. In a sense, we have 
validated this Model for change, and used it to 
substantially change our development processes 
and the customer’s view of our product 
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Topics 


• Introduction 

• Our Experiences 

• Results 

• Summary 

• Lessons Learned 
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Congratulations to the SEL! 


• Winner of the IEEE Computer Society Software 
Process Achievement Award for 1994 

• Motorola’s Cellular Systems Division (CSD) was 
“First Runner Up” 

• We are the “Avis” of Process Achievement this 
year, and “trying harder.” 
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Motorola Cellular 
Systems Division (CSD) 

• Approximately 1 ,000 in the R & D Division 

• Four locations: 

- Arlington Heights, IL, USA 

- Cork, Ireland 

- Tel Aviv, Israel 

- Ft. Worth, TX (the fourth country) 

• Data presented here is for the EMX 2500 
Switch Software Development Group 
(-300 staff) 
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CSD Key Events 


• Motorola has a corporate software engineer- 
ing goal to achieve SEI Level 3 by YE’95 

• CSD had first SEI Self-assessment in Nov.’90 

- Level 1 (are you surprised?) 

• Second Self-assessment, June ‘93 

- Level 2 (phew! Made it) 

• Third Self-assessment scheduled next week 
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CSD Key Strategy 
Decisions 




1 . Use SEI 5-level Model for “Requirements” 

2. Use IEEE 1074 for the “Design” 

3. Implement a “Process Improvement” 
Project 
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FIGURE 6 - Software Functionality (Delta KAELOC) 



^ Delta KAELOC 
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Lessons Learned 


• “Plan your work”--in this case Process 
Improvement 

• “Work your plan”-in this case the 
Process Improvement Project Plan 

• This Project has: 

- Requirements Specifications 

- Design Architecture 

- Implementation Phases 

- Verification and & Validation Phases 

^ 19th SEW J 

November 30, 1994 
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